What is HTTP Streaming?

HTTP Streaming is like an open line between the server and the client. Instead of sending a complete response and closing the connection, the server sends data continuously over an open connection. This allows the client to receive updates as soon as they are available, without polling the server repeatedly.

Think of it as being on a phone call where the server talks whenever there’s something new to share, and the connection stays alive until one side hangs up.

How HTTP Streaming Works

In a traditional HTTP request-response cycle, the client sends a request and the server responds with the full message before closing the connection. With HTTP Streaming, however, the server sends the response incrementally over time while keeping the connection open. This method enables real-time updates from the server to the client.

Chunked Transfer Encoding

One way to achieve HTTP Streaming is by using "chunked transfer encoding." With this technique, the server sends data in chunks without needing to know the full size of the response in advance. Each chunk is transmitted as soon as it’s ready, allowing real-time updates without introducing latency.

Server-Sent Events (SSE)

Another method, called "Server-Sent Events" (SSE), is a higher-level protocol specifically designed for streaming events from the server to the client. SSE allows the server to send updates in a structured format (text/event-stream) over a persistent connection. It is widely used for real-time notifications, logs, and other one-way communication where the server continuously pushes data to the client.

Practical Guide: Implementing HTTP Streaming

Let’s walk through two examples: one using Server-Sent Events (SSE) and another using chunked transfer encoding to stream log updates to the client. Each example will show you how to set up the server and client for real-time updates.

Using Chunked Transfer Encoding

Now, let’s create a server that streams log entries to the client using chunked transfer encoding:

Create a new project directory and initialize it:

    mkdir chunked-example
    cd chunked-example
    npm init -y

Install the required dependencies:

    npm install express

Create the server script (`server.js`):

const express = require('express');
const app = express();

app.use((req, res, next) => {
    res.setHeader('Access-Control-Allow-Origin', '*');  // Allow cross-origin for this demo
    next();
});

// Endpoint to stream logs using chunked transfer encoding
app.get('/stream', (req, res) => {
    res.setHeader('Content-Type', 'text/plain, charset=UTF-8');
    res.setHeader('Transfer-Encoding', 'chunked');

    const sendLog = (counter) => {
        res.write(`Log entry ${counter}\n`);
        if (counter < 10) {
            setTimeout(() => sendLog(counter + 1), 500);
        } else {
            res.end();
        }
    };
    sendLog(1);
});

app.listen(3000, () => {
    console.log('Chunked transfer encoding server listening on http://localhost:3000/stream');
});

Run the server:

node server.js

Client-side Implementation

Next, create an HTML page to connect to the server and display the log updates as they stream in using chunked transfer encoding:

<h1>Log Updates (Chunked Transfer Encoding)</h1>
<pre id='log'></pre>
<script>
fetch("http://localhost:3000/stream")
    .then(response => {
        const reader = response.body.getReader();
        const decoder = new TextDecoder();
        function readStream() {
            return reader.read().then(({done, value}) => {
                if (done) return;
                const logData = decoder.decode(value);
                document.getElementById("log").textContent += logData;
                return readStream();
            });
        }
        return readStream();
    })
    .catch(error => console.error("Stream error:", error));
</script>

This script fetches the log stream from the server and displays it in real time, appending each chunk of data as it arrives.

Using Server-Sent Events (SSE)

We’ll use Node.js to create a simple server that streams log entries to the client using Server-Sent Events:

Create a new project directory and initialize it:

mkdir sse-example
cd sse-example
npm init -y

Install the required dependencies:

npm install express

Create the server script (`server.js`):

const express = require('express');
const app = express();

app.use((req, res, next) => {
    res.setHeader('Access-Control-Allow-Origin', '*');  // Allow cross-origin for this demo
    next();
});

// Endpoint to stream logs using Server-Sent Events (SSE)
app.get('/events', (req, res) => {
    res.setHeader('Content-Type', 'text/event-stream');
    res.setHeader('Cache-Control', 'no-cache');
    res.setHeader('Connection', 'keep-alive');

    const sendLog = (counter) => {
        res.write(`data: Log entry ${counter}\n\n`);
        if (counter < 10) {
            setTimeout(() => sendLog(counter + 1), 500);
        } else {
            res.end();
        }
    };
    sendLog(1);
});

app.listen(3000, () => {
    console.log('SSE server listening on http://localhost:3000/events');
});

Run the server:

node server.js

Client-side Implementation

Now, create an HTML page to connect to the server and display the log updates as they stream in using SSE:

<h1>Log Updates (Server-Sent Events)</h1>
<pre id="log"></pre>
<script>
    const eventSource = new EventSource('http://localhost:3000/events');
    eventSource.onmessage = function(event) {
        document.getElementById('log').textContent += event.data + '\n';
    };
    eventSource.onerror = function(error) {
        console.error('SSE error:', error);
    };
</script>

This script connects to the SSE endpoint and displays log updates in real time as they are sent from the server.

Conclusion

HTTP Streaming provides a powerful way to deliver real-time updates from server to client. Both chunked transfer encoding and Server-Sent Events (SSE) are valid approaches for streaming data over HTTP, each with its own use cases. In this guide, we demonstrated how to implement both methods, helping you understand the options available for real-time data delivery.

FAQ